In this notebook I'll attempt to build a quick ML model (2-3 hours) to predict whether a mobile ad will be clicked given set of params. Data comes from the kaggle competition available in the follwing link: https://www.kaggle.com/c/avazu-ctr-prediction/overview/evaluation
We do have somewhat large test and training files. Zipped size of training data is over 1GB
! ls -lh | grep .gz
It's a good point to stop and discuss our strategy on processing files (data engineering) and training our ML models. Since we have somewhat large dataset we've got following options:
pdpipe is a good choice for lightweight data pipelineing option in pandas domain. Prefect which can be used for task scheduling and management (Great data pipelines) MLflow, MLLib ..etc), howevever it involves creating Spark clusters, which might be time consuming, considering this will be a quick task (2-3 hours) I tend to skip it for the time being. Let's have a brief look at the contents of train.gz and test.gz file
! gzcat train.gz | head | csvlook
! gzcat test.gz | head | csvlook
Looks like we need to predict click column
for convenience I'll extract the zipped files, to make it easier for dask to read
! gunzip test.gz && gunzip train.gz
! ls -lh | grep 'test\|train'
As expected uncompressed files are about 6x bigger than compressed versions
There are number of columns we can use to build our model, however before going any further, let's decide which columns we can use. There are number of things we can do here but I'd like do check the distribution of values briefly.My strategy would be:
pandas-profiling for a quick visualization# housekeeping
import pandas as pd
pd.set_option('display.max_columns', 30)
import pandas as pd
import dask.dataframe as dd
from pandas_profiling import ProfileReport
Use dask to read and sample data
from dask.distributed import Client
client = Client(n_workers=4, threads_per_worker=1, processes=False, memory_limit='2GB')
client
# read training lazily with dask
train = dd.read_csv('train', dtype={'id': 'float64'})
we've got ~ 40.4 million rows of training data
%%time
# sample with replacement (though with replacement doesn't really matter as the data set is big enough)
s_train = train.sample(frac=0.001, random_state=42).compute()
s_train.head()
s_train.to_pickle('data/ctr_data.pickle', compression='gzip')
Let's have a quick look at the columns to decide which ones to use in our ML model.
I'll use pandas-profiling to get a nice view
s_train = pd.read_pickle('data/ctr_data.pickle', compression='gzip')
report = ProfileReport(s_train, title='Train Data Report', explorative=True)
# Save to file for offline viewing
report.to_file('Initial_Report.html')
# view report inside Jupyter Notebook
# NOTE: If you are opening up this ipynb file in github the provfiling report won't be visible, please download repo and view the outputted html file to see report
report.to_notebook_iframe()
by looking at the pandas profiling output we can make few assumptions :
click has two values (0, 1), as expected and click (1) is not an rare event, roughly ~6/33. hour needs conversion, and we can create few categories out of hour. (format is YYMMDDHH, so 14091123 means 23:00 on Sept. 11, 2014 UTC.)C1 column has 7 different values though 3 most common values are responsible for more than 99% of. So we can filter to values (1005, 1002 and 1010). C1 seems to be correlated with device_typebanner_pos has 6 vals but (0 and 1) covers 99% of the casessite_id has high cardinality (1070 side-ids), and it's categorical variable, so it doesn't make much sense to use this (one hot encoding) to predict clicks. Discardedsite_domain also has high cardinality and doesn't make much sense to use as predictor. site_category has 18 unique values and 4 of them are responisble for 98% of the cases,(50e219e0, f028772b, 28905ebd, 3e814130)app_category 4 values are responsible for most of the cases : (07d7df22, 0f2161f8, cef3e649, 8ded1f7a, f95efa07)app_id, app_domain, device_id, device_ip, device_model all have high cardinalities, if we know device_ip we could have enriched it with geotag but encoded values it's not much of use. device_type can take 4 different values, and value 1 accounts for more than 92% percent of the cases. Probably not a very good predictor. Will think about it later. device_conn_type can take 4 vals and (0, 2 and 3) are responsible for 99.9% of the cases. c14 and c17 have high correlation in between, so it make sense to keep only one. However C14 or C17 looks like categorical variables with high cardinality so I'll drop these cols. c15 takes many values but (320, and 300) covers more than 99% of the cases. c16 takes many values but (50, and 250) covers 99% of the cases. Filter for those values. c18 takes 4 values and they all seem to be common (value 1 is the least common but appears in 7% of the cases). One hot encoding. c19 has high cardinality, skippingc20 looks like some ids, with 6 digits, such as 100084, but apart from 6 digit numbers there seems to be -1 in 46.6% cases. So we can potentially divide thist to -1 or others.
Based on the analysis above, I'll filter the sampled dataset:
cols_to_drop = ['id', 'site_id', 'site_domain', 'app_id', 'app_domain', 'device_id', 'device_ip', 'device_model', 'device_type', 'C14', 'C17', 'C19', 'C21']
def clean_data(df: pd.DataFrame, cols_to_drop: list) -> pd.DataFrame:
"""
Define all operations to make training + test data ready for prediction/model building
"""
# Drop columns those deemed unnecessary
df = df.drop(columns=cols_to_drop, axis=0)
# Reset sampled indexes
df = df.reset_index(drop=True)
# Convert hour field to datetime
df['date'] = pd.to_datetime(df['hour'], format='%y%m%d%H')
# and set the hour field to correct hour (Hour might have some predictive power on click through rate)
df['hour'] = df.date.dt.hour
# extract day of the week (maybe on weekends or some specific days people's habit is different)
df['day'] = df.date.dt.day_name()
# C1 conditions
c1_conds = {1005, 1002, 1010}
df['C1'] = df['C1'].apply(lambda x: x if x in c1_conds else -1)
# banner_pos conditions
banner_pos = {0, 1}
df['banner_pos'] = df['banner_pos'].apply(lambda x: x if x in banner_pos else -1)
# site category conditions
site_cats = {'50e219e0', 'f028772b', '28905ebd', '3e814130'}
df['site_category'] = df['site_category'].apply(lambda x: x if x in site_cats else 'others')
# app_category conditions
app_cats = {'07d7df22', '0f2161f8', 'cef3e649', '8ded1f7a', 'f95efa07'}
df['app_category'] = df['app_category'].apply(lambda x: x if x in app_cats else 'others')
# C15 conditions
c15_conds = {320, 300}
df['C15'] = df['C15'].apply(lambda x: x if x in c15_conds else -1)
# C16 conditions
c16_conds = {50, 250}
df['C16'] = df['C16'].apply(lambda x: x if x in c16_conds else -1)
# C20 conditions
df['C20'] = df['C20'].apply(lambda x: False if x == -1 else True)
# No need to keep duplicates
df = df.drop_duplicates().reset_index(drop=True)
return df
cleaned_train = clean_data(s_train, cols_to_drop=cols_to_drop)
Let's run the profiling again to verify our operations
report = ProfileReport(cleaned_train, title='Cleaned Train Data Report', explorative=True)
# first save to disk for offline viewing
report.to_file('Cleaned_Report.html')
# view inside jupyter notebook (If you cannot see any report here, open up 'Cleaned_Report.html' from disk)
report.to_notebook_iframe()
Note: The following code assumes we work on unzipped (
train.gz) and sampled dataset. Since I can only spare no more than 3 hours on this assignment I decided to work ona smaller memory fitting data for testing and training
AdaBooostClassifier . I chose AdaBoost as it by default uses DecisionTrees (which are easy to interpret but prone to overfitting) and ensemble methods like AdaBoost deals with overfitting issues and generally have better prefictive capabilities than base models. GridSearchCV to find optimum paramsimport pandas as pd
from sklearn.metrics import make_scorer, fbeta_score, accuracy_score
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.ensemble import AdaBoostClassifier
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report, accuracy_score, f1_score
# Helper functions, configs
from beeswax_assignment.wrangling import clean_data
from beeswax_assignment.constants import cols_to_drop, one_hot_encoded_cols, SRC_DATA_PATH, f_BETA
def load_data(ctr_data):
"""
Load ctr_data and create feature and label vectors
Args:
ctr_data (Path): filepath where samped data is located
Returns:
X (pandas.DataFrame): Dataframe, used as features
Y (numpy.Array): Numpy array. Used as labels (click column)
label_names (list): label names
"""
# read sampled data from disk
df = pd.read_pickle(ctr_data, compression='gzip')
# clean the data (based on exploratory analysis)
df = clean_data(df, cols_to_drop=cols_to_drop)
# apply one hot encoding for the categorical columns
df = pd.get_dummies(df, columns=one_hot_encoded_cols).reset_index(drop=True)
# Target label
y = df['click'].to_numpy()
# Features
X = df.drop(['click', 'date'], axis=1)
# feature names
feature_names = list(X.columns)
return X, y, feature_names
def build_model(scorer):
"""
Build machine learning pipeline
Args:
scorer: sklearn.metrics.make_scorer: What score (f-Beta, accuracy..etc) should
we optimize for
Returns:
sklearn.model_selection.GridSearchCV: ML model (and pipeline) to build ML model
using GridSearchCV and AdaBoostClassifier
"""
pipeline = Pipeline([
('clf', AdaBoostClassifier(random_state=42))
])
parameters = {
'clf__n_estimators': [10, 30, 50, 100, 200],
'clf__learning_rate': [0.5, 1., 10]
}
# Use GridSearchCV to find optimum hyperparameters for ML model
model = GridSearchCV(pipeline, param_grid=parameters, scoring=scorer)
return model
def evaluate_model(model, X_test, y_test):
"""
Evaluate model performance
Args:
model (sklearn.model_selection.GridSearchCV): Trained ML model
X_test (pandas.DataFrame): Test feature set, here these are tweets.
y_test (pandas.DataFrame): Test label set
Prints:
str: classification preport and accuracy scores
Returns:
None
"""
# Classify the tweets using test set
y_pred = model.predict(X_test)
print("Classification report: \n",classification_report(y_test, y_pred))
print("Accuracy score: {}".format(accuracy_score(y_test, y_pred)))
print("f1 score: {}\n\n".format(f1_score(y_test, y_pred, average='macro')))
# Load data
X, y, feature_names = load_data(SRC_DATA_PATH)
# Split the 'features' and target:'click' data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 42)
# Create scorer
scorer_accuracy = make_scorer(accuracy_score)
print('Building high accuracy model...')
model_accuracy = build_model(scorer_accuracy)
print('Training high accuracy model...')
model_accuracy.fit(X_train, y_train)
print('Evaluating high accuracy model...')
evaluate_model(model_accuracy, X_test, y_test)
best_model = model_accuracy.best_estimator_['clf']
print('\nOptimized params for High accuracy model is {}'.format(best_model))
important_features = best_model.feature_importances_
features_sorted = pd.Series(data= important_features, index=feature_names).sort_values(ascending=False).head(10)
print('Top 10 important features for high accuracy model is \n\n{}'.format(features_sorted))
scorer_fbeta = make_scorer(fbeta_score, beta=f_BETA)
print('Building high recall model...')
model_beta = build_model(scorer_fbeta)
print('Training high recall model...')
model_beta.fit(X_train, y_train)
print('Evaluating high recall model...')
evaluate_model(model_beta, X_test, y_test)
best_model = model_beta.best_estimator_['clf']
print('\nOptimized params for High recall model is {}'.format(best_model))
important_features = best_model.feature_importances_
features_sorted = pd.Series(data= important_features, index=feature_names).sort_values(ascending=False).head(10)
print('Top 10 important features for high recall model is \n\n{}'.format(features_sorted))
Though this was a very quick model building excercise, there are many aspects are disregarded here since I'm only restricted to 3 hours to complete this assignment.